Control and optimization of indirect evaporative cooling units for data center cooling

ABSTRACT

Embodiments of the present invention provide control solutions for data center thermal management and control using an IDEC system. The data center is arranged in a cold air room wall supply and hot air room wall return configuration. The sensors, such as temperature sensors and/or pressure sensors, are utilized to measure and record thermal data. The data is then processed and used to control the IDEC system to adjust operating conditions to satisfy dynamic thermal requirements of the data center. The control functions include: 1) controlling the IDEC system to adjust cooling modes and operating conditions as needed to maintain proper data center thermal environment; and 2) identifying different optimal operating conditions and parameters for the IT room and IDEC system.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to data centers.More particularly, embodiments of the invention relate to controllingand optimizing indirect evaporative cooling units for data centers.

BACKGROUND

Heat removal is a prominent factor in computer system and data centerdesign. The number of servers deployed within a data center has steadilyincreased as server performance has improved, thereby increasing theamount of heat generated during the regular operation of the servers.The reliability of servers used within a data center decreases if theenvironment in which they operate is permitted to increase intemperature over time. A significant portion of the data center's poweris used for removing the heat generated by the electronics packagedwithin the server. As the number of servers within a data centerincrease, a greater portion of the power is commensurately consumed bythe data center to cool electronic components within the servers.

Indirect evaporative cooling/cooler (IDEC) is one of the popular coolingsolutions for data centers. It can be understood as an economizationsolution which uses outside air or liquid to cool the data center airthrough air-to-air or liquid-to-air heat exchangers. In addition, IDECuses evaporative cooling when the outside air dry-bulb temperature isnot sufficiently low, in which it turns into a wet-blub temperaturerunning mode. Direct expansion (DX) cooling is used during extremeambient temperature conditions, such as hot summer days.

The control of an IDEC unit is critical for both cooling system and datacenter system reliability and energy efficiency. For IDEC unit, theautomation involves the controlling of an internal blower/fans speed, anexternal blower/fans speed, an evaporative pump (e.g., on and off,speed), and DX operating condition. A control system is to adjust theoperating conditions of the IDEC unit to maintain proper thermalenvironment in the data center information technology (IT) room. Thedata center IT room is under dynamic conditions most of the time. Thethermal environment includes supply air temperature, supply airhumidity, supply air flow rate/velocity and supply air quality.Maintaining these parameters within proper envelops are critical forserver and IT normal operating and long-term reliability. The controlbecomes more critical when there are multiple IDEC units used in thesystem.

Most of existing solutions may not applicable to the actual data centerbuilding and room. The actual data center and building design aredifferent from case to case. The function of the control system ismaintaining and ensuring the thermal conditions including airtemperature and air flow rate satisfying the requirements. Variations inany of these conditions may result in an impact on the other. Theconventional solutions may not be a reliable and efficient one for thecase described in this document.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 shows a perspective view of a data center system according to oneembodiment.

FIGS. 2A-2B shows a top view of a data center system according tocertain embodiments.

FIG. 3 shows a top view of a data center system according to anotherembodiment.

FIG. 4 shows an example of a data center room configuration according toone embodiment.

FIG. 5 shows an example of a data center configuration according toanother embodiment.

FIG. 6 is a block diagram illustrating an example of a control systemaccording to one embodiment.

FIG. 7 is a flow diagram illustrating a thermal management process of adata center system according to one embodiment.

FIG. 8 is a flow diagram illustrating a thermal management process of adata center system according to another embodiment.

FIG. 9 is a flow diagram illustrating a thermal management process of adata center system according to another embodiment.

FIG. 10 shows examples of operating modes of an IDEC system according toone embodiment.

FIG. 11 is a flow diagram illustrating a thermal management process of adata center system according to another embodiment.

DETAILED DESCRIPTION

Various embodiments and aspects of the inventions will be described withreference to details discussed below, and the accompanying drawings willillustrate the various embodiments. The following description anddrawings are illustrative of the invention and are not to be construedas limiting the invention. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentinvention. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present inventions.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

Embodiments of the present invention provide control solutions for datacenter thermal management using an IDEC system. The data center isarranged in a cold air room wall supply and hot air room wall returnconfiguration. The sensors, such as temperature sensors and/or pressuresensors, are utilized to measure and record thermal data. The data isthen processed and used to control the IDEC system to adjust operatingconditions to satisfy dynamic thermal requirements of the data center.The control functions include: 1) controlling the IDEC system to adjustoperating conditions as needed to maintain proper data center thermalenvironment; and 2) identifying different optimal operating conditionsand parameters for the IT room and IDEC system. The thermal managementoperations and optimization include monitoring and controlling cold airroom (supply) thermal conditions and hot air room (return) thermalconditions, using cold air room and/or hot air room thermal data ascontrol signals for the IDC systems; reducing the impact of variationsin the IDEC system on the cooling air conditions supplied to the ITroom; proper control design for the actual application scenarios;operating conditions optimization of the IDEC system; and reducing theimpacts of hot air recirculation, operator activities on the roomdynamics.

According to one aspect of the invention, a data center system includesan IT room to contain a number of electronic racks. Each electronic rackmay contain a number of server blades and each server blade may containone or more IT components (e.g., processors, memory, storage devices)that may generate heat during the operations. The data center systemfurther includes a cold air room and a hot air room connected to the ITroom. The cold air room is configured to receive cold air of an internalairflow from an IDEC unit and to distribute the cold air to theelectronic racks of the IT room to exchange heat generated from the ITcomponents of the electronic racks. The cold air room includes a firstset of one or more temperature sensors to sense and measure thetemperature of the cold air room. The hot air room is configured toreceive hot air carrying the exchanged heat from the IT room and toreturn the hot air back to the IDEC unit. The cold air room and the hotair room may be located on opposite sides of the IT room to allow theinternal airflow to flow from one side to the opposite side, travelingthrough the airspace of the electronic racks.

The data center system further includes a control system communicativelycoupled to the cold air room, the hot air room, and/or the IT room, aswell as the first set of temperature sensors. In response to sensor datareceived from the temperature sensors of the first set, the controlsystem is configured to analyze the sensor data based on a first set ofthermal management rules. The control system transmits one or morecontrol signals to the IDEC unit to modify one or more operatingparameters of the IDEC unit based on the analysis to provide an optimalthermal environment for operating the IT components of the electronicracks. For example, the control system may configure the IDEC to operatein one of the operating modes, such as, a dry air cooling mode, anevaporative cooling mode, and/or a DX (or air conditioning) mode.

The control system may further adjust one or more operating parametersin each of the operating modes such as fans speed, volume of mist ofliquid drops, and/or condenser and evaporator settings. In oneembodiment, the hot air room further includes a second set oftemperature sensors to sense and measure the temperature of the hot airroom. The control signals may be generated based on the sensor data ofboth the first set and the second set of temperature sensors. The IDECmay be controlled or adjusted based on the average temperatures measuredby the temperature sensors or the temperature difference between thecold air room and the hot air room.

According to one embodiment, the data center further includes a set ofpressure sensors disposed at least within the cold air room or betweenthe interface between the cold air room and the IT room to sense andmeasure the air pressure of the cold air entering the IT room. Based onthe temperature sensor data, the control system may adjust the openingratio of the louvers (e.g., windows, doors) between the cold air roomand the IT room to control the airflow volume entering the IT room,which in turn adjusts the temperature of the IT room. Similarly, a setof pressure sensors may also be disposed within the hot air room or onthe interface between the IT room and the hot air room for similarpurposes. Based on the temperature information and/or the air pressureinformation, the control system can control the IDEC unit to operate indifferent operating modes by adjusting the operating parameters of theIDEC unit, such that the IT room can operate in an optimized thermalenvironment.

According to another aspect of the invention, a data center systemincludes an IT room to contain a number of electronic racks. Eachelectronic rack can contain a number of server blades and each serverblade can contain one or more IT components operating therein. The datacenter system further includes a cold air room near a first side of theIT room and a hot air room near a second side of the IT room. The coldair room is configured to receive cold air from an IDEC unit and todistribute the cold air to the electronic racks via one or more louversof airflow windows disposed on a wall between the cold air room and theIT room, to exchange heat generated from IT components of the electronicracks. The hot air room is configured to receive hot or warmer aircarrying the exchanged heat from the IT room and return the hot/warmerair back to the IDEC unit.

The data center system further includes a first set of one or morepressure sensors disposed near the cold air room or between the cold airroom and the IT room (e.g., near the louvers) to sense and measure theair pressure of the internal airflow entering the IT room from the coldair room. The data center system further includes a control systemcoupled to the cold air room, the IT room, the hot air room, and thepressure sensors. In response to sensor data received from the pressuresensors, the control system is configured to analyze the sensor databased on a set of thermal management rules or algorithms and to adjustan opening ratio of at least one of the louvers interfacing the cold airroom and the IT room based on the analysis to adjust a cold airflowvolume entering the IT room through at least one of the louvers, whichin turn adjusts the thermal conditions of the IT room.

According to one embodiment, a second set of one or more pressuresensors may be disposed near the hot air room or between the hot airroom and the IT room (near the airflow opening on a wall between the ITand hot air rooms). The control system is to control the opening ratioof the louvers further based on the sensor data obtained from the secondset of pressure sensors. The pressure sensor data of the second set mayalso be utilized to control the opening ratio of louvers disposedbetween the IT room and the hot air room. The opening ratios of thelouvers may be adjusted based on the averaged air pressures measured bythe pressure sensors or the pressure difference between the cold airroom and the hot air room. The purpose of adjusting the louvers is toensure uniform air flow rate delivery to the room from the walls.

According to a further aspect of the invention, an optimization processmay be performed to determine an optimal setting of an IDEC unit for adata center system, such that the electronic racks can operate in anoptimal thermal environment, and at the same time, using minimum coolingpower. The optimization process is an iterative process using a set ofthermal setting candidates of the IDEC. Initially, an optimal partialpower usage effectiveness (PPUE) is configured as an initial PPUE value.An optimal set of parameters of an IDEC unit is set to an initial set ofparameters. For each of the operating condition candidates of IDEC unit,the IDEC unit is configured based on a set of parameters associated withthe current operating condition candidate. The pressure sensor data iscollected from the pressure sensors disposed at various locations withina cold air room connected to an IT room having a number of electronicracks. Each electronic rack contains a number of IT components. Thepressure sensors are configured to measure the air pressure of the coldair received from the IDEC unit (e.g., internal airflow). The cold airis to be distributed to the electronic racks to exchange heat generatedfrom the IT components and transformed into hot/warmer air. The hot airis then received by the hot air room and returned back to the IDEC unitfrom the hot air room.

In addition, according to one embodiment, the temperature sensor data iscollected from the temperature sensors disposed at various locationsnear the cold air room. A current thermal condition (e.g., temperature,air pressure) of the IT room is determined based on the pressure sensordata and temperature sensor data. It is then determined whether thecurrent thermal condition satisfies a predetermined thermal conditionassociated with the IT room. A current PPUE of the data center system iscalculated and compared with the optimal PPUE. If the current PPUE islower than the optimal PPUE value, the optimal PPUE value is replaced bythe current PPUE value. The set of parameters of the IDEC unit isdesignated as the optimal set of the parameters for the IDEC unit. Afterall of the operating condition candidates of the IDEC unit have beenprocessed, the IDEC unit is then configured base on the optimal set ofparameters, such that the IT room is to operate in an optimal thermalcondition, for the current IT workload (heat load) condition.

FIG. 1 shows a perspective view of a data center system according to oneembodiment. Referring to FIG. 1, data center system 100 includes an ITroom 101, a cold air room 102, and a hot air room 103. IT room 101includes a number of electronic racks such as electronic racks 111-112.Each electronic rack contains one or more IT components arranged in astack. An IT component can be a computer server providing data servicesto clients. Alternatively an IT component can be a peripheral device ora network appliance device such as cloud storage systems. Each ITcomponent may include one or more processors, memory, and/or a storagedevice that may generate heat during operations. The electronic racksare arranged in a number of rows of electronic racks, in this example,rows 104-105 of electronic racks. The rows of electronic racks arearranged spaced apart to form one or more cold aisles and one or morehot aisles. In this embodiment, although there are only two rows 104-105of electronic racks shown, there can be more rows to be contained in ITroom 101.

In one embodiment, each row of electronic racks is positioned orsandwiched between a cold aisle and a hot aisle. In this example, row104 and row 105 are positioned apart from each other to form cold aisle114A, hot aisle 115, and cold aisle 114B. Hot aisle 115 is formedbetween row 104 and row 105. Row 104 is positioned or sandwiched betweencold aisle 114A and hot aisle 115, while row 105 is positioned orsandwiched between cold aisle 114B and hot aisle 115. In one embodiment,hot aisle 115 is contained or enclosed by hot aisle containment (orcontainer or other enclosures). In another embodiment, the cold aislesare contained in a containment environment instead of the hot aisles. Ina further embodiment, both hot aisles and cold aisles are contained inan enclosed environment. In one embodiment, the backend of theelectronic racks of rows 104-105 are facing hot aisle 115, while thefrontends of the electronic racks are facing cold aisle 114A or coldaisle 114B and away from hot aisle 115.

In one embodiment, cold air room 102 is located and adjacent to a firstside of IT room 101, while hot air room 103 is located and adjacent to asecond side of IT room 101. In this example, the first side and thesecond side are opposite sides of IT room 101. Cold air room 102 isconfigured to receive cold air or cool air via one or more inlet portsfrom a cold air source such as cold air source 180. The cold air isallowed to enter IT room 101 from cold air room 102 via one or moreopenings disposed on the wall between cold air room and IT room 101 (notshown). The cold air entering IT room 101 to form cold aisles 114A-114B.

Hot air room 103 is configured to exhaust the hot air from hot aisle 115and return the hot air or warmer air back to the cold air source 180 forheat exchange. Note that cold air source 180 can include a heatexchanger or chiller. For example, cold air source 180 can be an IDECsystem or device, which can operate in a number of different operatingmodes (e.g., air cooling mode, evaporative cooling mode, and DX coolingmode). Alternatively, cold air source 115 can simply be the naturalambient air outside of the data center system 100.

An evaporative cooler is a device that cools air through the evaporationof water. Evaporative cooling differs from typical air conditioningsystems, which use vapor compression or absorption refrigeration cycles.Evaporative cooling works by exploiting water's large enthalpy ofvaporization. The temperature of dry air can be dropped significantlythrough the phase transition of liquid water to water vapor(evaporation). Direct evaporative cooling is used to lower thetemperature and increase the humidity of air by using latent heat ofevaporation, changing liquid water to water vapor. In this process, theenergy in the air does not change. Warm dry air is changed to cool moistair. The heat of the outside air is used to evaporate water. Indirectevaporative cooling is a cooling process that uses direct evaporativecooling in addition to some type of heat exchanger to transfer the coolenergy to the supply air. The cooled moist air from the directevaporative cooling process never comes in direct contact with theconditioned supply air.

Referring back to FIG. 1, in this example, the cold air is received fromone or more inlets or inlet ports disposed on a wall of cold air room102, where the wall is substantially parallel with a third side of ITroom 101. The third side of IT room 101 is substantially perpendicularto the first side and the second side, while the first side and thesecond side are substantially parallel to each other. Similarly, the hotair is exhausted from hot air room 103 to the external environment orback to cold air source 180 via one or more outlets or outlet portsdisposed on a wall of hot air room 102, where the wall is substantiallyparallel with the third side of IT room 101.

According to one embodiment, hot aisle 115 is enclosed or containedwithin hot aisle containment 120, such that the hot air cannot escape orspill from hot aisle 115 into other areas of IT room 101 such as coldaisles 114A-114B. Instead, the hot aisle enters hot air room 103 fromhot aisle via one or more openings (e.g., windows, doors) disposed on awall between hot aisle 115 and hot air room 103. In one embodiment, theopenings allow an operator or a user to enter hot aisle 115 from hot airroom 103 to access the backend of the electronic racks, for example formaintenance services. Doors are needed on the 102 and 103, for operatorsto be able to enter cold air room 102 and hot air room 103. And no dooris needed on hot aisle 115, and doors are needed on the walls betweencold air room 102 and IT room 101, according to some embodiments.Similarly, the openings disposed on the wall between cold air room 102and IT room 101 may include one or more doors to allow an operator oruser to enter cold aisles 114A-114B from cold air room 102. As a result,entering or leaving cold aisles or hot aisles would not have asignificant impact on the cold air distribution and hot air exhaustion.That is, entering or leaving hot aisle 115 would not alter cold airdistribution for cold aisles 114A-114B, because hot aisle 115 isseparated from cold aisles 114A-114B and the rest of IT room 101 by hotaisle containment 120. Similarly, entering or leaving cold aisles114A-114B would not affect hot air exhaustion of hot aisle 115, sinceopening a door for a user to enter or leave would not mix the cold airand the hot air.

FIG. 2A shows a top view of a data center system according to oneembodiment. The data center system can represent a top view of datacenter 100 of FIG. 1. Referring to FIG. 2A, the electronic racks arearranged into a number of rows similar to rows 104-105 of FIG. 1. Therows of electronic racks are positioned and spaced apart from each otherto form cold aisles 114A-114C (collectively referred to as cold aisles114) and hot aisles 115A-115D (collectively referred to as hot aisles).Hot aisles 115A-115D are each enclosed or contained in a hot aislecontainment structure. Cold air room 102 is positioned adjacent to afirst side of IT room 101. Hot air room 103 is positioned adjacent to asecond side of IT room 101.

In this example, the first side and the second side of the IT room 101are opposite sides. The cold air is received from an external cold airsource such as an IDEC system via one or more inlet or intake ports 201into cold air room 102. The cold air then enters IT room 101 via one ormore openings (e.g., windows, doors with louvers) 211-213 and enter coldaisles 114A-114C. The cold air then enters from the frontends of theelectronic racks, travels through the airspace of the electronic racks,and enters hot aisles 115A-115D. As described above, hot aisles 115 areconfigured as an enclosed or contained environment that can receive theair flows from cold aisles through the airspace of the electronic racks.The hot air is prevented from spilling into other areas of IT room 101.The hot air can only exit IT room 101 and enter into hot air room 103via openings 221-224.

In one embodiment, at least one of the openings 211-213 disposed on awall between cold air room 102 and IT room 102 includes a door to allowan operator or user to enter IT room 101 and cold aisles to access theelectronic racks such as frontends of the electronic racks formaintenance services. Similarly, each of openings 221-224 (no doors)allows an operator or user to enter the corresponding hot aisle toaccess the backend of the electronic racks adjacent to the hot aisle.

In this embodiment, the cold air enters into cold air room 102 via inlet201 from the third side (e.g., the right hand side) of IT room 101. Thehot air leaves hot air room via outlet 202 of IT room 101 from the thirdside. The third side is different from the first side adjacent to coldair room 102 and the second side adjacent to hot air room 103. In oneembodiment, the first side and the second side are substantiallyparallel to each other. The third side is substantially perpendicular tothe first side and/or the second side. Alternatively, according toanother embodiment as shown in FIG. 2B, the cold air enters cold airroom 102 from a side parallel to the first side. The hot air leaves hotair room 103 from a side parallel to the second side.

In the configurations as shown in FIGS. 2A-2B, the cold air is directlyinjected into cold air room 102 from external ambient environment withfiltering. The hot air is exhausted from hot air room 103 back to theexternal environment. According to another embodiment, cold air room 102and hot air room 103 are coupled to a cooling system such as an IDECunit for heat exchange via respective airflow channels coupling the IDECunit with the cold air room 102 and hot air room 103. The externalcooling system may be configured as a modular system that can be coupledto cold air room 102 and hot air room 103, where the cooling system maybe provided by a third party entity that is different than an entityproviding or maintaining the data center system. Similarly, cold airroom 102 and/or hot air room 103 may be configured as a modular module,which can be provided by a third party entity. Cold air room 102, hotair room 103, and/or cooling system 401 may be provided by the same ordifferent vendors or organizations.

FIG. 3 is a block diagram illustrating a top view of a data centersystem according to one embodiment. Referring to FIG. 3, theconfiguration of IT room 101, cold air room 102, and hot air room 103 issimilar to the one as shown in FIG. 2A. In addition, an IDEC unit 310 iscoupled to cold air room 102 and hot air room 103, where IDEC unit 310may represent the cooling system 180 of FIG. 1. In this embodiment, IDECunit 310 provides cold air to cold air room 102 via a cold air channeland receives hot air from hot air room 103 via a hot air channel,forming an internal airflow that is circulated through cold air room102, IT room 101, and hot air room 103.

In addition, according to one embodiment, a number of sensors such assensors 301A-301D (collectively referred to as sensors 301) may bedisposed at various locations. Sensors 301 may include temperaturesensors and/or pressure sensors. The temperature sensors are configuredto monitor and measure the temperatures of the airflow, while thepressure sensors are configured to monitor and measure the air pressureof the internal air flow. In this embodiment, a first set of sensorssuch as sensors 301A-301B are disposed on a wall between cold air room102 and IT room 101. For example, the sensors may be disposed on theopening such as airflow windows or doors. In one embodiment, the sensorsmay be disposed on the louvers mounted on the wall between cold air room102 and IT room 101, as shown in FIG. 4. Similarly a second set ofsensors such as sensors 301C and 301D may be disposed on the wall orlouvers between IT room 101 and hot air room 103.

Furthermore, a control system 300 is communicatively coupled to IT room101, cold air room 102, hot air room 103, sensors 301, and IDEC unit310. According to one embodiment, control system 300 receives sensordata or signals 305 from sensors 301 and performs an analysis on thesensor data to determine the current thermal environment or condition ofthe data center. Based on the analysis, control system 300 transmits oneor more control commands or signals 306A-306B (collectively referred toas control signals or commands 306) to control or adjust certainoperating parameters or settings of the data center unit (e.g., IT room101, cold air room 102, and hot air room 103) and IDEC unit 310. Thecontrol system 300 is to maintain proper and stable thermal conditionsin the cold air room 102 and hot air room 103. A stable thermalenvironment of cold air room 102 and hot air room 103 ensures a stablethermal environment in the IT room 101. In addition, another goal is toimprove energy efficiency of the IDEC system 310 by optimizing theoperating conditions of IDEC system 310.

FIG. 5 is a block diagram illustrating an example of an IDEC systemaccording to one embodiment. Referring to FIG. 5, a typical IDEC unitcan operate in a number of operating modes, including 1) an air coolingmode; 2) an evaporative cooling mode; and 3) a DX mode similar to airconditioning. Any one or more of these operating modes can be activatedor deactivated individually and/or simultaneously. When IDEC 300operates in an air cooling mode, one or more external fans 501 areutilized to circulate and exhaust external airflow 512 out, while one ormore internal fans 504 circulates the internal air flow 511 across thechamber of IDEC 300 without directly contacting external airflow 512 forheat exchange. The fans speed of fans 501 and 504 can be controlled andadjusted by control system 300. When IDEC 310 operates in an evaporativecooling mode, one or more evaporative pumps 502 are utilized to pump andspray mist of liquid drops onto the external airflow 512 to reduce thetemperature of the external airflow 512, which the heat carried byexternal airflow 512 may cause the mist of liquid drops to evaporate,which in turn lowers the temperature of the external airflow. When IDECunit 310 operates in a DX mode, the DX system 503 is invoked to reducethe temperature of external airflow 512 and/or internal airflow 511. Thecontrol system 300 can turn on or turn off any one or more of thecooling modes based on the sensor data received from various temperaturesensors and/or pressure sensors according to a set of thermal managementrules or algorithms.

FIG. 6 is a block diagram illustrating an example of a control systemaccording to one embodiment. Referring to FIG. 6, control system 300includes, but is not limited to, analysis module 601, louver controller602, internal fans controller 603, external fans controller 604,evaporative pump controller 605, and DX controller 606. Modules 601-606may be implemented in software, hardware, or a combination thereof. Forexample, modules 601-606 may be loaded into a memory and executed by oneor more processors to perform various operations. Note that some or allof modules 601-606 may be integrated into fewer modules or a singlemodule.

Analysis module 601 is configured to monitor and receive sensor datafrom most or all of the sensors of the data center system, and toperform an analysis on the sensor data to determine the current thermalconditions or environment of the data center. Analysis module 601 mayperform the analysis based on a set of one or more thermal managementrules or algorithms 610 to determine a set of one or more actions toconfigure and mange IDEC unit 310, for example, by invoking specificmodules 602-606.

Louver controller 602 is configured to control the louvers (e.g., intakelouvers) between the cold air room 102 and IT room such as louvers211-213, as well as the louvers (e.g., exhaust louvers) disposed betweenIT room 101 and hot air room 103 if there is any. A louver can becontrolled by louver controller 602 to be more opened, closed, ordifferent opening angles, etc. By adjusting the opening and/or angle ofa louver, the volume of the airflow flowing through the louver can becontrolled. In one embodiment, based on the analysis, analysis module601 determines a set of one or more controlling parameters and invokeslouver controller 602 to adjust an opening ratio of a particular louver.

Referring back to FIG. 4, there are various pressure sensors disposed ondifferent locations of different louvers 211-213. Depend upon thedirection of the internal airflow, the air pressures on differentlouvers may be different, which may represent different volumes of theairflow flowing through the corresponding louvers. For example, if theair pressure detected by pressure sensor 301A is higher than the airpressure detected by pressure sensor 301B, it may indicate that theairflow volume flowing through louver 211 is higher than the airflowvolume flowing through louver 212. As a result, the opening ratio oflouver 211 may need to be adjusted lower (e.g., to be closed) to allowless air flowing through louver 211, and more air flowing through louver212. The goal of managing louvers 211-213 is to provide relatively evenairflow volumes flowing through all louvers.

According to one embodiment, analysis module 601 receives the pressuresensor data from most or all pressure sensors disposed on louvers211-213 and determines the air pressures measured by the pressuresensors. Analysis module 601 calculates an average air pressure(P_(ave)) based on the measured air pressures as follows:

$P_{ave} = {\frac{1}{n}{\sum\limits_{1}^{n}P_{n}}}$where P_(n) represents an individual air pressure measured by aparticular one of the pressure sensors. The opening ratio of thecorresponding louver can be adjusted based on the difference between theindividually measured air pressure and the average air pressure:ΔP=P _(n) −P _(ave)

In the cases that the static air pressure within the cold air room is indifference, the corresponding louvers will be controlled to eitherincrease the opening ratio (low pressure) or decrease the opening ratio(high pressure). Thus, if the pressure difference ΔP is positive, itmeans the local air pressure is relatively high compared to the averageair pressure and the corresponding louver needs to be opened less toreduce the air pressure.

An opening ratio of a louver refers to the opening angle percentage ofthe louver. An opening ratio of 0 means the louver is fully closed, inwhich an airflow moving path is fully blocked. An opening ratio of 100%means the louver is fully open, in which the airflow can pass throughthe louver with the minimum flow resistance. A louver can be consideredas an airflow valve, similar as liquid fluid valve opening ratio.

The ΔP is calculated to capture how much is the difference among eachindividual P_(n), by comparing to the Pa. In addition, a predeterminedtolerance range for difference (e.g., +/−0.01 pounds per square inch orPSI) is utilized. If the ΔP is within the acceptable range (e.g.,+/−0.01 PSI), there is no need to take any action. If ΔP>+range (e.g.,0.01 PSI), this means the corresponding local location static pressureis high. This means the louver needs to introduce more resistance byreducing the opening ratio to make the value lower. This means less airflow flowing through. Similarly, if ΔP<−range (e.g., −0.01 PSI), theopening ratio needs to be increased.

For example, there a total of five sensors, data collected are 1.2, 1.3,1.5, 1.2, and 1.3 PSI. Then the averaged value P_(ave) is 1.3. If +/−0.1is the acceptable range, then the value 1.5 is out of the acceptablerange. The other four are within the range. Then ΔP=1.5−1.3>0.1(acceptable range), the louver open ratio should be decreased. Whenlouver is decreased, more resistance is introduced, the pressure dropbecomes larger, then the static pressure reading value will be decreasedas a result.

In one embodiment, if there are multiple pressure sensors disposed on aparticular louver, the air pressure of that particular louver may becalculated based on the average of the pressure readings from thepressure sensors associated with the louvers.

In one embodiment, if there are multiple pressure sensors disposed onboth the supply and return sides of each louver, the air pressure dropacross each louver may be calculated and used for controlling the louveropening ratio, similar principle as using the static pressure.

FIG. 7 is a flow diagram illustrating a process of controlling louversbased on sensor data according to one embodiment. Process 700 may beperformed by processing logic which may include software, hardware, or acombination thereof. For example, process 700 may be performed bycontrol system 300. Referring to FIG. 7, in operation 701, one or morepressure sensors are utilized to monitor and measure air pressures ofthe louvers. In operation 702, an average air pressure Pa is calculatedbased on the pressure readings form the pressure sensors. For each ofthe louvers, in operation 703, a pressure difference ΔP between the airpressure of the louver P_(n) and the average pressure Pa, is calculated,i.e., ΔP=P_(n)−P_(ave).

In operation 704, it is determined whether the pressure difference ΔP iswithin a predetermined range (e.g., an acceptable range such as +/−0.01PSI). If so, it means the air pressure of the louver is normal.Otherwise, if the pressure difference ΔP is outside of the predeterminedrange, in operation 705, it is determined whether the pressuredifference ΔP is greater than either the upper limit of the range orless than the lower limit of the range (e.g., −0.01). If ΔP is greaterthan the upper limit of the range, in operation 706, the opening ratioof the louver is decreased. If ΔP is lower than the lower limit of therange (e.g., −0.01), the opening ratio of the louver is increased inoperation 707. Note that process 700 may be performed further in view oftemperature sensor data obtained from the temperature sensors disposedin the cold air room, the hot air room, and/or the IT room. That is, thelouvers may be controlled and adjusted further based on the temperaturesensor data representing the temperature of the cold air room and/or hotair room.

According to another aspect of the invention, the pressure sensor datacan also be utilized to control the fans speed of the internal fans suchas fans 504 of FIG. 5. For example, the air pressure of the internalairflow is proportionally related to the temperature in the IT room 101.By measuring the air pressure of the internal airflow, the temperatureof the IT room can be derived. Thus, based on the air pressure of theinternal airflow, the speed of the internal fans 504 can be adjust,which in turn adjust the thermal condition of the IT room 101.

FIG. 8 is a flow diagram illustrating a process of controlling aninternal airflow according to one embodiment. Process 800 may beperformed by processing logic which may include software, hardware, or acombination thereof. For example, process 800 may be performed bycontrol system 300. Referring to FIG. 8, in operation 801, pressuresensor data is collected from the pressure sensors disposed at the coldair room 102 and hot air room 103. In operation 802, the air pressure ofthe cold air room and the air pressure of the hot air room aredetermined, and the difference between the cold air room air pressureand the hot air room air pressure is calculated. In operation 803, it isdetermined whether the pressure difference is within a predeterminedrange. If the pressure difference is not within the predetermined range,in operation 804, it is determined whether the air pressure of the coldair room is higher than a first predetermined set point. If the airpressure of the cold air room is below the first predetermined setpoint, in operation 805, it is determined whether the pressuredifference between the cold air room and the hot air room is below asecond predetermined set point (e.g., upper limit or lower limit of therange). If so (e.g., below a lower limit of the range), in operation806, the internal fans speed is increased; otherwise (e.g., above anupper limit of the range), the internal fans speed is decreased inoperation 807. The first and second predetermined set points may thesame or different. Note that process 800 may be performed further inview of temperature sensor data obtained from the temperature sensorsdisposed in the cold air room, the hot air room, and/or the IT room.That is, the internal fans may be controlled and adjusted further basedon the temperature of the cold air room and/or hot air room.

According to a further aspect, supply air temperature data at the coldair room 102 are recorded and used as a part of the control signals ofIDEC operating conditions and modes. The cold air room temperatures canbe considered as the cold aisle temperatures, as well as the supply airinlet temperatures of the servers. This control includes the IDECexternal blower such as fans or blowers 501, different operating modesincluding dry-blub operating, wet-bulb operating and DX cooling mode.Normally, the data center supply air temperature is predefined. Thereare also industry guidelines for setting data center cold aisle airtemperature, such as ASHRAE (American Society of Heating, Refrigerating,and Air-Conditioning Engineers) thermal guidelines for data centers.T_(max) is the maximum temperature value collected. The required rangesare the acceptable temperature ranges for allowing the cold air roomtemperature to stay at. Since there are multiple operating modes can beadjusted on IDEC 300 to adjust cooling capacity, including changing IDECexternal blower speed, evaporative pump speed, and DX operating mode.Such actions can be selectively performed or in parallel in response tothe sensor data collected from the temperature sensors.

Referring back to FIG. 6, according to one embodiment, in response totemperature sensor data received from the temperature sensors, analysismodule 601 is configured to analyze the temperature sensor data and toinvoke external fans controller 604 to adjust the fans speed of fans orblowers 501 to control the volume or speed of external airflow 512.

FIG. 9 is a flow diagram illustrating a process of controlling fans foran external airflow based on temperature sensor data according to oneembodiment. Process 900 may be performed by processing logic which mayinclude software, hardware, or a combination thereof. For example,process 800 may be performed by control system 300. Referring to FIG. 9,in operation 901, the internal fans of an IDEC unit is set to an initialspeed and the server blades of the IT rooms start to operate inoperation 902. In operation 903, the temperature of the cold air room isdetermined based on temperature sensor data obtained from thetemperature sensors disposed near or within the cold air room. Inoperation 904, the temperature sensor data is analyzed, includingcalculating the average temperature of the temperature sensors, etc. Inoperation 905, it is determined whether the temperature (e.g., theaverage temperature) of the cold air room is within a predeterminedrange. If the temperature is outside of the predetermined range, inoperation 906, it is determined whether the temperature is higher than apredetermined set point (e.g., an upper limit of the range). If so, inoperation 907, the external fans speed is increased. In addition, theevaporative pumps 502 and/or DX system 503 may be activated. Otherwiseif the temperature is lower than a predetermined set point (e.g., alower limit of the range), in operation 908, the external fans speed maybe decreased, and the evaporative pumps 502 and/or DX system 503 may bedeactivated.

As described above, an IDEC unit can operate in a number of operatingmodes, including an air cooling mode, an evaporative cooling mode, and aDX mode. Each of these operating modes can be individually activated ordeactivated, in sequence or in parallel as shown in FIG. 10. Inaddition, the specific operating parameters such as fans speed, mistliquid drop volume, and the settings of the condenser and evaporator ofthe DX system may also be individually configured. According to anotheraspect, the operating modes and operating parameters of these subsystemsof an IDEC unit can be configured to find an optimal thermal environmentor condition specifically tailored to a particular data center systemgiven the specific configuration of the data center system, such asworkload and layout. Note that process 900 may be performed further inview of pressure sensor data obtained from the pressure sensors disposedin the cold air room, the hot air room, and/or the IT room.

Referring back to FIG. 6, analysis module 601 can perform an iterativeprocess to find a set of operating parameters or operating modes for theIDEC unit 310 that provides an optimal thermal operating environment forthe IT room 101. For example, during the iteratively process, inresponse sensor data received from the sensors, analysis module 601performs an analysis on the sensor data to determine the current thermalcondition of the IT room 101. Analysis module 601 then invoke internalfans controller 603, external fans controller 604, evaporative pumpcontroller 605, and DX controller 606 to control and adjust therespective operating parameters of the air cooling subsystem, theevaporative cooling subsystem, and the DX subsystem of IDEC unit 310.Such an optimization process is iteratively performed until all of thecombinations of the operating modes and parameters have been processed.One of the combination settings of the subsystems is then selected thatcan satisfy the required thermal condition (e.g., lowest temperature) toIT room, and at the same time, minimize the water and electricalconsumption.

FIG. 11 is a flow diagram illustrating an optimization process fordetermining an optimal cooling unit operating conditions for a datacenter at certain IT workload condition according to one embodiment.Process 1100 may be performed by processing logic which may includesoftware, hardware, or a combination thereof. For example, process 1100may be performed by control system 300. Referring to FIG. 11, inoperation 1101, an IDEC unit is configured according to the initialsettings and the electronic racks are turned on in operation 1102. Inoperation 1103, the sensor data is collected from the sensors such astemperature sensors and pressure sensors. In operation 1104, the sensordata is analyzed to determine the operating environment of the datacenter such as temperature and power consumption, etc. A partial powerusage effectiveness (PPUE) value is set to an initial optimal PPUEvalue. Operation 1105 is to designed for safety operating. Before thesystem optimization process, it needs to be ensured that the IT roomthermal environment requirement is satisfied.

Power usage effectiveness (PUE) is a ratio that describes howefficiently a computer data center uses energy; specifically, how muchenergy is used by the computing equipment (in contrast to cooling andother overhead). An ideal PUE is 1.0. Anything that is not considered acomputing device in a data center (i.e. lighting, cooling, etc.) fallsinto the category of facility energy consumption.

Referring back to FIG. 11, in operation 1105, it is determined whetherthe thermal conditions satisfy the IT room requirement, where thepredetermined condition may be compatible with the industry standard fora particular type of data center or a particular IT workload condition.If the thermal condition does not satisfy the IT requirement, inoperation 1106, the IDEC unit is configured to be set to the maximumcooling capacity condition (e.g., the thermal environment with thelowest possible temperature), and the above operations 1103-1105 areiteratively performed. The rationale behind is that we would ensure thedata center is operating in an acceptable thermal environment beforeoptimizing the operating parameters of the IDEC unit.

Once the IT thermal condition has been satisfied in operation 1105, theIDEC unit operating conditions are adjusted in operation 1107 for eachof the operating condition candidates in operation 1108 (e.g., conditioncandidates as shown in FIG. 10). In operation 1109, it is determinedwhether the IT thermal condition is still satisfied. If not, a nextoperating condition or a next set of operating parameters of the IDECunit is selected and the above operations 1107-1108 are iterativelyperformed. If the IT thermal condition has been satisfied in operation1109, the new or current PPUE value is calculated based on the currentsettings of the IDEC unit in operation 1110. In operation 1111, it isdetermined whether the current PPUE value is lower than the optimal PPUEvalue. If so, in operation 1112, the optimal PPUE value is updated andset to the current PPUE value of the current iteration. In addition, thecurrent set of operating modes or operating parameters of the IDEC unitare designated as an optimal set of operating modes or operatingparameters for the IDEC unit, which may be stored in a persistentstorage device such as a hard disk of control system 300. The aboveoperations are iteratively performed until all of the operatingmode/parameter candidates (e.g., candidates as shown in FIG. 10) havebeen processed in operation 1113.

Once the optimal set of operating parameters of the IDEC unit has beendetermined via the above iterative processes, the IDEC unit can then beconfigured according to the optimal set of operating parameters. In oneembodiment, process 1100 may be performed periodically (e.g., duringinitial start or reboot of the data center) to find the optimal set ofoperating parameters at the point in time, as the operating environmentof the data center, such as the workload, may change from time to time.

In this design, PPUE which is a metrics for cooling energy consumptionare considered as the criteria for the optimal process, in anotherembodiment, water usage effectiveness, or combined water usageeffectiveness and PPUE can be used as alternative criterias.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the disclosure also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures may beperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

Embodiments of the present disclosure are not described with referenceto any particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof embodiments of the disclosure as described herein.

In the foregoing specification, embodiments of the invention have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A data center system, comprising: an informationtechnology (IT) room to contain a plurality of electronic racks, eachelectronic rack having a plurality of IT components therein; a cold airroom adjacent to a first side of the IT room to receive cold air of aninternal airflow from an indirect evaporative cooling (IDEC) unit and todistribute the cold air to the electronic racks to exchange heatgenerated from the IT components, wherein the cold air room includes afirst set of at least one or more temperature sensors to sense atemperature of the cold air room; a hot air room disposed adjacent to asecond side of the IT room to receive hot air carrying the heatexchanged from the electronic racks and to return the hot air to theIDEC unit; and a control system coupled to at least a portion of the atleast one or more temperature sensors of the first set, wherein thecontrol system is configured to in response to sensor data from thetemperature sensors of the first set, analyze the sensor data based on afirst set of thermal management rules wherein analyzing the sensor datacomprises iteratively adjusting at least one or more operatingparameters and determining whether the first set of thermal managementrules are satisfied, and transmit at least one or more control signalsto the IDEC unit to modify the at least one or more operating parametersof the IDEC unit based on the analysis to provide an optimal thermalenvironment for operating the IT components of the electronic racks. 2.The data center system of claim 1, wherein the hot air room comprises asecond set of temperature sensors to sense a temperature of the hot airroom, wherein the control system is configured to analyze sensor datareceived from the temperature sensors of the second set, and wherein theat least one or more control signals are generated further base on theanalysis on the second set of temperature sensors.
 3. The data centersystem of claim 1, wherein in analyzing the sensor data, the controlsystem is configured to calculate an average temperature of the cold airroom based on temperature data obtained from the temperature sensors ofthe first set, compare the average temperature with a first predeterminetemperature threshold, and in response to determining that the averagetemperature is greater than the first predetermined temperaturethreshold, the control system is configured to increase a fans speed ofa first set of at least one or more fans of the IDEC unit responsiblefor circulating an external airflow to exchange heat with the internalairflow received from the hot air room.
 4. The data center system ofclaim 3, wherein the control system is further configured to activate anevaporative pump of the IDEC unit to spray liquid onto external air tolower a temperature of the external airflow.
 5. The data center systemof claim 3, wherein the control system is further configured to activatea direct expansion (DX) system of the IDEC unit to lower a temperatureof the external air flow.
 6. The data center system of claim 3, whereinthe control system is further configured to compare the averagetemperature with a second predetermined temperature threshold, anddecrease the fans speed of the at least one or more fans of the firstset, in response to determining that the average temperature is belowthe second predetermined temperature threshold.
 7. The data centersystem of claim 6, wherein the control system is further configured todeactivate an evaporative pump of the IDEC unit to stop spraying liquidonto external air.
 8. The data center system of claim 6, wherein thecontrol system is further configured to deactivate a direct expansion(DX) system of the IDEC unit.
 9. The data center system of claim 1,wherein the cold air room further comprises a first set of at least oneor more pressure sensors to sense an air pressure of the internalairflow in the cold air room, and wherein the control system isconfigured to adjust a distribution of the internal airflow entering theIT room from the cold air room based on air pressure data obtained fromthe pressure sensors of the first set.
 10. The data center system ofclaim 9, wherein the hot air room further comprises a second set ofpressure sensors to sense an air pressure of the internal airflow in thehot air room, and wherein the distribution of the internal airflow isadjusted further based on air pressure data obtained from the pressuresensors of the second set.
 11. The data center system of claim 10,wherein the control system is configured to modify a fans speed of asecond set of at least one or more fans of the IDEC unit to adjust aflow volume of the internal airflow based on the air pressure dataobtained from the first set and second set of pressure sensors.
 12. Thedata center system of claim 9, wherein the control system is configuredto calculate an average pressure of the internal airflow based onpressure data obtained from the pressure sensors, compare the averagepressure against a predetermined pressure threshold, and adjust anopening ratio of a louver interfacing the cold air room and the IT roomto adjust a flow volume of the internal airflow entering the IT roombased on the comparison.
 13. The data center system of claim 12, whereinthe louver comprises a plurality of louvers and the first set ofpressure sensors comprises a plurality of subsets of pressure sensors,each of the subsets of pressure sensors being associated with one of thelouvers, wherein each subset of pressure sensors is utilized to measurea flow volume of the internal airflow flowing through a correspondinglouver.
 14. A method of thermal management of a data center system, themethod comprising: receiving, by a control system, sensor data from aplurality of temperature sensors disposed within a cold air roomadjacent to an information technology (IT) room of a data center system,the IT room including a plurality of electronic racks, wherein the coldair room is configured to receive cold air from an indirect evaporativecooling (IDEC) unit and to distribute the cold air to travel through anairspace of the electronic racks to exchange heat generated from theelectronic racks to reach an hot air room adjacent to the IT room,wherein the hot air room is configured to return hot air carrying theexchanged heat back to the IDEC unit; performing an analysis on thesensor data based on a first set of thermal management rules, andtransmitting one or more control signals to the IDEC unit to modify oneor more operating parameters of the IDEC unit based on the analysis toprovide an optimal thermal environment for operating IT components ofthe electronic racks.
 15. The method of claim 14, wherein the hot airroom comprises a second set of temperature sensors to sense atemperature of the hot air room, wherein the control system isconfigured to analyze sensor data received from the temperature sensorsof the second set, and wherein the one or more control signals aregenerated further base on the analysis on the second set of temperaturesensors.
 16. The method of claim 14, wherein performing an analysis onthe sensor data comprises: calculating an average temperature of thecold air room based on temperature data obtained from the temperaturesensors of the first set; comparing the average temperature with a firstpredetermine temperature threshold; and increasing a fans speed of afirst set of one or more fans of the IDEC unit responsible forcirculating an external airflow to exchange heat with the internalairflow received from the hot air room, in response to determining thatthe average temperature is greater than the first predeterminedtemperature threshold.